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ABSTRACT 

Three groups of students were exposed to the same com- 
puter-administered programmed instruction in numerical 
differentiation with different degrees of access to the 
output from the typewriter terminal* Analysis of co- 
variance showed no significant difference on posttest 
scores between students who were allowed to keep the out- 
put and those who were not # nor between those students 
who could look back during the session at previous output 
and those whose view was restricted to the most recent 
output m 
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Chapter I 
THE PROBLEM 

1 „ 1 Overview of Computer-Assisted Instruction 

Llol Modes of Use 

The application of the relatively new technology of com- 
puters to the ancient problem of learning has taken many 
formSo These can be classified, on the bases of purpose and 
means of achieving that purpose, in five distinct categories, 

(1) As a student learning tool, the computer is used 
for problem solving* The student may write programs 
or use prewritten programs which are at his disposals 
This mode is characterized by the control which the \jf 
student exercises over the use of the tool 0 t 

(2) As a teaching aid, the computer can be used in \ 
a manner similar to chalkboard, slides, films 0 etc* 
The ability to access a computer from a classroom 
terminal allows an instructor to give examples which 
may otherwise be omitted due to time constraints or 
tedious computations* This permits demonstration of 9 
for instance 0 convergence of an iterative process 0 
repeated calls of a recursive function, or the effect 
of various reentry speeds on the landing site of a 
space vehicle,, 



(3) As a simulation device, the computer can provide 
students an opportunity to apply principles and con- 
cepts learned earlier. The most familiar example is 
the Sumerian game, in which the student assumes the 
role of the King of Sumer. The actions he takes to 
solve various social , political , and economic crises 
determine the; future of the country, 
i: (4) As a tool for instruction management, the computer 
is used for storage and retrieval of teaching materials 
and student record Individual student lessons can 
be prescribed by analysis of past performance. 
{5) The most commonly held image of computer use in 
the instructional process is that of a student seated 
at a terminal, being taught by a program over which he 
has little or no control. This author-controlled mode 
is used for drill, testing, and the presentation of 
new material. 

1*1.2 Costs 

The above modes of computer use share some common prob- 
lems, not the least of which is cost. There seem to be four 
main costs involved: central processing unit (CPU), termi- 
nals, communications r and sof tware* Regardless of whether 
the computer is owned or leased, or CPU time is bought from 
someone else, the cost is very real. Terminal costs vary 
widely depending upon the type of terminal 9 and the mode of 
use determines the number required. Certainly s to involve a 
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significant number of students in simulation, author--* 
controlled, or problem sol ving mode requires multiple ter- 
minals,, Terminal costs are discussed more fully in Section 
L3„8o Economic considerations may require one CPU to be 
accessed by widely scattered terminals^ thereby introducing 
the cost of communication among the several locations. 
Depending upon the number of terminals,, their geographic 
distribution., and the means of communication, the per- 
student-hour operating cost of such a system is likely to 
be less than that incurred using multiple independent com- 
puters,, The currently most common means of communication 
is via commercial telephone lines* Finally, there is the 
cost of buying,, leasing or writing the software necessary 
to utilize the hardware. This includes the system support 
software as well as the instructional programs. As an 
indication of the magnitude of the cost of the latter, it 
is variously estimated that 100-200 man -hours are necessary 
to prepare one hour of author-controlled . instruction [4„ 
ppo 85-89] * 

Llo3 NB£i'-2ili2£ the Scope 

The focus of this investigation is the author-con- 
trolled mode,, All subsequent references to computer- 
assisted instruction (CAI) should be interpreted as author- 
controlled presentation of new material to a single student 
at a terminal, Feldhusen and Szabo [15 J give a goocJ survey 
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of the many studies which have been made of learning charac- 
teristics,, branching strategies, methods of reinforcement, 
stimulus presentation, and other program' dependent features 
of CAT „ This m.udy is concerned with the effect of charac- 
teristics of the terminal it&elf upon learning, 

l„2 Common features of terminals 

i « Z„ 1 CPU Requirements 

To be of use in CAT u a terminal must be able to access 
a central processor., The demands it places upon the CPU 
vary with the type of terminal and its major use* For 
example, an unbuffered volatile cathode ray tube must be 
continually refreshed by the CPU* The CPU may be dedicated 
entirely to the CAI system or, as is often the case,, may 
operate interactively with the student user while doing 
batch processing in the backgroundo 

1 * 2 o 1 Keyboards 

Human- to- computer communication i «> achieved in most, 
instances via a "keyboard,, similar to that of an ordinary 
typewriter o The keys,, used singly or in combination^ repre- 
sent alphabetic v numeric,, or special characters* Among the 
special characters are such symbols as + @ and #, and con- 
trol characters which govern the operation of the terminal 
and the CPU* As discussed in Section L3o6 y many video dis- 
play terminals have other input mechanisms. 
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1.2.3 Output Display Media 

The computer- to-human communication can take many forms, 
some generated by the computer 9 others simply under computer 
control. 

The vast majority of these forms are visual in nature, 
although some work has been done using random access tape 
recorders, under computer control, to provide audio stimula- 
tion. 

The display media required on a terminal depend to a 
certain extent upon the intended use„ For instajnce, the 
ability to provide audio output would be desirable in 
teaching a foreign language and nearly imperative in dealing 
with very young children,, but of questionable use in teach- 
ing numerical analysis to college student s 0 Similarly, 
devices capable of producing high resolution graphic -dis- 
plays are not needed if the only required output is text. 

It appears, then, that if one has need of a specialized 
terminal , knows the capabilities required / and is willing to 
spend the necessary money f one can obtain the terminal* 
Most CAI practitioners are not in that position* They either 
want a general-purpose terminal h are not sure what their 
needs are, operate under a tight budget B or, quite likely, 
fit all three descriptions. The problem is then reduced to 
a choice between two basic types of terminal, the typewriter 
style jsr.c che cathode ray tube* 



1-3 Tynewr :< tor ter m inals versus CaJ^ode rube Term^n^ijs 

1 . 3 . 1 Definitions 

Let us consider a "standard 1 CUT terminal l.o .i r.c.l ucie a 
display screen^ alphanumeric keyboard J and light peno inter- 
ence to CRT's will imply all terminals cf this type, ;One 
such terminal is the CC-30 Display Station from Computer Com- 
munications 0 Inc 0 ) Reference to TWR*s will imply the broad 
class of typewriter-like terminals 9 including the Teletype 
ASR models and the IBM 2741c Comparing the two types, we 
note the following differences 0 

1 • 3o 2 Speed 

The physical movement of the typing element limits the 
print speed of TWR»s to 10-15 characters per second 0 whereas 
the display rate of a CRT is limited in practice only by the 
transmission rate* When one uses low speed communication 
lines the two types of display occur at approximately the 
same rate* Wcdtke and Gilman [16] suggest that a display 
rate significantly slower than a student's reading rate can 
produce boredom and subsequent avoidance of CAT as a means 
of learning* Typical- reading rates for college students 
(300-400 words per minute) are two and one-half to three 
times the print rate of TWR" So 

1 e 3 * 3 Noise 

The noise level of a CRT is negligible, whereas a TWR 
produces harsh $ staccato sounds caused by carrier returns, 
case shifts, type element rotations, and printing impact., 

ERLC 
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The noise problem is particularly acute when the terminal is 
used in a small room with poor acoustics or when many ter- 
minals are located in one roortio 

1,3.4 Reliability 

The typing mechanism of a TWR is el ectr omecbanical 9 
whereas the display production, in a CRT is electronic. The 
susceptibility of the moving parts to maladjustment and 
damage makes the TWR less reliable than a CRT in terms of 
both f requency-of -repair and correct display of characters 
[3% In both types of terminals, the keyboard used for in- 
put is electromechanical 0 but this is a much less frequent 
source of trouble than the output mechanism 0 

1 o 3 „ 5 Graphics Capability 

In order to plot a graph on a TWR terminal., one chooses 
one of the available characters and places that character in 
selected print positions. The spacing of the type element- 
over many blank positions causes a potentially annoying de- 
lay. Moreover, the characters can be no closer than one line 
apart vertically and/or one print posit j on horizontally. 
This may be entirely acceptable in representing discrete 
date but offers only a crude approximation to continuity * 

Most CRT's, on the other hand, produce an alphanumeric . 
character by illuminating selected points within a rectangu- 
lar dot matriXo If the points can be illuminated individual- 
ly , as some terminals allow, rather than as part of a 
character 0 they can be used for producing graphs 3 maps, and 
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diagrams of greater resolution than is possible with a TWR„ 
Moreover the rapid display presents the picture to a user as 
a conceptual entity rather than a series of isolated charac- 
ters* This graphic capability allows the CAI author a wider 
choice of subject matter and presentation mode* 

1.3o6 Response Mode 

To use a TWR terminal, a student must, at the very least, 
be able to find a given letter on the keyboard § strike the 
key, and strike the return key* Thus the non- typist, 
especially the young child, is penalized in terms of time 
spent entering responses. The insecurity and distraction 
associated with the "hunt and peck" method may interfere with 
the student's concentration on the subject matter 5 

On the typical CRT the electron-detecting light pen can 
be used to select, by merely pointing, a response from a 
multiple choice list displayed on the screen, Swets [15] 
has used the CRT to provide a two dimensional interpretation 
to responses which were linear on a TWR G (See Section l 0 5 a l«) 
Combined with the graphics capabilities discussed in the pre- 
vious section, the light pen greatly increases the variety of 
teaching methods available to CRT users 0 

lo3*7 Error C orrection 

On a TWR, a typographical error is overstruck, corrected 
on the line below, or "erased" by typing a special character 
once for each character entered after the one in error* 
There may be a negative effect upon the user when the in- 
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correct character remains in place , On a CRT, when the 
offending character is overstruck, the original character is 
replaced by the most recently entered character „ 

1.3.8 Cost 

As would be expected , the cost of a terminal depends 
upon a multitude of factors. A CRT of the type under con- 
sideration can be purchased for $6000- $9000, The variance 
in price is due to features such as storage buffers, 
character sets, display size and resolution^, display modes 
(e.g. graph or character) 9 etc, 

Within the TWR classification there is also a choice of 
features, such as maximum line width , interchangeable type 
elements, special character sets ff print speed, noise, etc, 
A Teletype Model 33 ASR can be purchased for less than 
$1000, whereas an IBM 2741 may cost $4000, 

The importance of the cost factor is reflected in the 
statement by Dr. Patrick Suppes in 1966 that " * * „ we'll 
probably be using mostly teletype and typewriter stations 
for the next five years for the simple reason of economics 1 * 
[13]. 

103.9 Hard Copy 

TWR's produce a record on paper of a student 1 s entire 
session at a terminal, while CRT's in general produce no 
such record* The following section deals with this hard 
copy and its use» 



lo4 The R o1-e of Har d Copy 
lo4*l During the Session 

Because of the transitory nature of the display on a 
CRT t the student does not have a record of his preceding 
work. Assuming a sequehtial presentation of subject matter 0 
it may be instructive for the student to review, at any time 
he wifijhes, some particular point or the over* all development 
of the lesson* With hard copy output this can be done simply 
by looking back at the pertinent sections of the printed 
record* With a CRT, unless special measures are employed g 
the record is lost when it is removed from the display 
screen,, Several possible means of overcoming this diffi- 
culty are discussed in Section 1,4«3, 

1 0 4 0 2 After the Session 

Upon leaving a session at a TWR a the student can take 
along a complete sequential record of the lessen, including 
material presented, requests for input 5 his own responses, 
program reaction to his responses, data summaries, etc* 
Depending upon the subject matter, the student v s age, the 
instruction mode, and various external factors, this record 
may be used in any of several ways 0 

(a) Comparison of the output with other students 0 may 
give an understanding of why certain responses were incorrect* 
Comparison of results on quizzes administered during the 
session may provide the peer approval or competitive urge 
needed to encourage a particular student 0 



(b) Parental inspection of the output could increase 
parents' understanding of the nature of computer-assisted 
instruction Such inspection may also prove to be valuable 
incentive to students to perform wello 

(c) One of the greatest potential benefits of CAI is 
individualization of instruction*. Utilizing data on an 
individual student fl s past performance, the teaching program 
could generate a homework assignment tailored to the partic- 
ular student °s needs 0 This could be printed on the hard 
copy output and taken home 0 

(d) Perhaps the most likely and most valuable use of 
hard copy is as a study guide for the student himself o Time 
constraints at the terminal often prevent the student from 
studying the development of the lesson as deeply as he would 
like c There may also be specific points which the student 
did not understand and which he would study further if he 
had a record of the lesson at that pointo If no printed 
record of the lesson is available, further study of the 
lesson itself is not likely 0 

(e) As one means of further study, the student could 
take the output to an " authority" , be it the teacher involved 
or some one else, for a conference concerning the material 
presented in the lessono The hard copy would be especially 
valuable in the case that the 11 authority" was not familiar 
with the lesson e 

(f) Aside from student use, the printed record could 



be used as a diagnostic aid by the teacher who could 
review the student's entire terminal session on a reduced 

time scale at the teacher's convenience^ The CAI program 

author could use output from the student, sessions to look 

for program errors, extremely difficult parts,, ambiguities „ 

etc* 

1 * 4 « 3 Alt erhatives 

Some of the needs for hard copy mentioned above can be 
met by a partial record of the session 0 This can be pro- 
vided in several ways other than having a TWR print the 
entire session,, 

When a CRT is used, the program author is aware that 
hard copy will not be created and can alter his program 
accordingly* Important points and summaries car) be stored 
and made available for display upon student request. It is 
conceivable that enough information could be kept to re- 
create the session later 9 but this would require vast 
amounts of storage and its value is not clear* 

Another possibility is use of a terminal which supplies 
both the advantages of a CRT and a hard copy mechanism* 
Many CRT's in the sophistication and price range under con- 
sideration can be equipped optionally with printers, The 
cost naturally depends upon the speedy quality 0 and versatil- 
ity of the printer* The as one example, can incor- 
porate a Teletype 33 Read Only Printer that sells for approxi- 
mately $600 o Other terminals are available which are 



basically CRT's, but which can produce g by thermal or photo- 
graphic means, a hard copy reproduction of the image current- 
ly on the screen. 

The question then arises as to what to print .. This 
choice could be made by the program author , who knows the re- 
lationships which exist between various parts of the program 
and who may speculate on what portions the student, may want 
printed. Or the choice could be left to the student g who 
knows what he wants but may not always know what he needs * 

One further possible solution is to store on disk what- 
ever amount (possibly all) of the session one desires to list 
then have all session records printed on a common printer a 
The listings could then be distributed physically to the 
students or teacher. Such an approach is taken with the IBM 
2848 Display Control system, in which the controller can 
handle multiple 2260 Display Stations and one 1053 Model 4 
Printer. 

Of the possible uses of hard copy specified in Section 
lo4.1 and Section 1.40 2^ one sees that these hybrid devices 
can satisfy some better than others* The greatest diffi- 
culty lies in anticipating what use will be made of the hard 
copy and recording the pertinent portion of the session*, 
Nothing short of printing the entire session will serve all 
the needs completely. 



1 . 5 Definition of the Problem 

1 . 5 • 1 Related Research 

As mentioned earlier, many comparative studies have 
been made, with widely varying results, comparing computer- 
assisted instruction, programmed text, and classroom lecture* 
Other investigators have compared instructional strategies , 
such as remedial vs. forward branching,, differing degrees of 
feedback (prompting), and varying amounts of student control 
over the lesson sequence. These studies have been directed 
toward devising an optimal instructional program and are not 
directly related. to the study herein described* 

Relatively few investigators have addressed themselves 
to the question of machine-to-student communication, Glaser, 
Lipson and Ramage [6] have attempted to characterize the in- 
terface between student and subject matter 0 an interface 
\>rtiich naturally includes the physical communication devices. 
Their major concern is the specification of techniques for 
optimal utilization of the visual and auditory senses of stu- 
dents. Consequently, their recommendations for developing 
devices for maximum flexibility fall primarily in the "long 
range" category, 

Wodtke and Gilman [16] found that college students 
working in a programmed text and students exposed to the 
same sequence administered by computer through a typewriter 
terminal did not achieve significantly different scores on 
a posttest. The computer group did take considerably longer 
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to complete the lesson. The investigators concluded that 
the typewriter print speed was slower than the students 1 

reading rate, suggesting that the typewriter may be more 

appropriate for relatively nonverbal subject matter 0 

Johnson and Borman [8], taking their cue from Wodtke 
and Gilman, exposed 90 college students to a CAI lesson in 
basic physics under one of four presentation modes s audio, 
display, type, slide. In type mode, the typewriter terminal 
was the sole interface device « In each of the others, the 
typewriter was used for student entries and some information 
display. Lengthy displays were presented with photographic 
slides, a pre-printed workbook, or audio tape messages. No 
significant difference was found in either posttest scores 
or completion times of the four treatment groups* 

Swets and others [14] were concerned with teaching sub- 
jects to identify sounds which could vary in frequency v 
amplitude, percent of on-t^lme, interruption rate, and length. 
Their experience led them to believe that the typewriter 
might be an inconvenient means of communication. In later 
studies [15] dual experiments were conducted, a typewriter 
being used in one and a CRT in the other 0 Each sound was 
"named" by a five digit number, each digit representing on 
a 1-5 scale the value of one of the five dimensions listed 
above. In the TWR experiment, the student had to name the 
sounds he heard by typing in this five digit number. Com- 
parison of incorrect and correct names required comparison 
of two numbers. In the CRT version of the experiment the 
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naming was done by porting a light pen at intersections of 
a 5 X 5 jrid matrix whose horizontal units wers the five 
dimensions and whose vertical units were the values of each 
dimension. Thus a spatial interpretation was established 
between, for instance, high pitch and higher points of the 
matrix . Comparison of incorrect responses with correct 
names was done by matching x's and o 9 s on the matrix* This 
apparently more convenient means of response and feedback 
did not produce significantly better results on the post- 
treatment test, 

1.5.2 Focus of t he Study 

One of the most urgent tasks in CAI research is to 
determine the circumstances under which each of the many 
modes of machine- to-student communication is effective. Of 
the studies cited above, only the last involved the two most 
widely employed types of CAI devices 0 the typewriter and the 
cathode ray tube terminals . Even in Swet. °s work the 
terminal was of seemingly minor importance 0 since the infor- 
mation presented was codified and always in the same formate 

The original idea for this study was to compare a type- 
writer terminal (the Datel Thirty-21„ which is very similar 
to an IBM 2741) and a CRT terminal (the CC-30 from Computer 
Communications, Inc.) as the sole interface with a student 
in administering programmed instruction,. When one looks at 
the differences enumerated in Section 1«3«, however, it is not 
difficult to imagine that some have little bearing on reten~ 



tion of subject matter* Cost, for example, is a concern of 
those responsible for providing the interface but not a 
factor directly involved in the instructional process* 
Except for an annoyance factor, reliability is in much the 
same category. The other dif f erences, with the exception 
of the lack of hard copy, are favorable toward the CRT* * 
Thus if cost is not a major obstacle, the only potential 
reason for not using CRT's is the absence of hard copy out- 
put. 

The author therefore decided to analyze the value of 
hard copy, both during and after the terminal session* 
Because of the interests of the sponsoring organization and 
the environment in which the investigation was to be con- 
ducted, the study was focused upon the teaching of numerical 
analysis to upper level college undergraduates and beginning 
graduate students. The instructional mode chosen 0 for 
reasons stated in Section 3o4«l f was computer- administered 
programmed instruction with limited prompting available 
upon student request* 



Chapter II 
THE STUDY 

2»1 Procedure 

Twenty-three students at the upper undergraduate- lower 
graduate level were divided randomly into three test groups. 
All subjects were given a pretest designed to measure their 
knowledge of the specific subject to be presented and their 
grasp of the mathematical ideas upon which the presentation 
was based. Each subject was exposed to the same computer- 
administered programmed instruction sequence teaching 
numerical differentiation at a typewriter terminal . Members 
of the control group, hereinafter referred to as Group 1, 
used the terminal in the normal manner and were instructed 
to take the printed record of the session with them upon 
leaving. Members of Group 2 operated the terminal in the 
same manner but were not allowed to remove the printed out- 
put from the terminal room. For members of Group 3 & the 
terminal was fitted with a box-like device which allowed 
the user visual access to only the most recent part of the 
output. Such a view is similar to what one would see on a 
CRT screen. (Details of the physical arrangements are 
given in Section 2.5.) Two days after the last student 
underwent treatment, a posttest was given. Analysis of 
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covariance was used to determine whether the group scores 
were significantly different,, Similar tests, of secondary 
interest, were performed to determine whether the groups 
varied significantly in other respects o 

2 0 2 Subjects 

2 o 2 * 1 Char acteristics 

The subjects were all enrolled in the course COMP 150, 
Introduction to Numerical Methods, offered by the Department 
of Computer and Information Science of the University of 
North Carolina at Chapel Hillo The course is taken by some 
beginning graduate students in the same department as partial 
fulfillment of the requirements for a Master of Science 
degree* Other students are primarily graduates or advanced 
undergraduates majoring in the physical sciences who are 
interested in applying numerical techniques within their 
discipline* Table 2„1 shows the academic fields of interest 
of the subjects by group; Table 2 Q 2 shows the academic year 
in which the" subjects were classified' Table 2 Q 3 gives a 
sketchy profile of the subjects 0 academic backgrounds in 
mathematics* The data shown in the tables were gathered by 
. a questionnaire given to the students at the time of the 
posttesto 

Some qualitative observations can be made on the com- 
position of the groups. Group 2 may be characterized as 
being further advanced in their studies and, by the authors 

ERLC 
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observations 9 older than the other subjects., The same group 
appears to differ from the others in area of major interest* 
Especially notable is the absence of Computer and Information 
Science majors and the presence of three biological scien- 
tists. One might reasonably predict that the members of 
Group 2, having been in their field of specialization longer 
and therefore farther removed from their formal mathematics 
training, would have more difficulty with the lesson, From 
data gathered by the teaching program itself 9 one finds that 
only 42% of Group 2 had used a computer terminal before 9 
compared with 50% and 63% of Groups 1 and 3, respectively* 

It should be noted here that of the 23 subjects 0 two did 
not return the questionnaire , two others did not take the pre- 
test, and one did not take the posttest* All statements in 
this report referring to a group should be interpreted as 
applying to that subset of the group for which data are 
available. 



Computer Science 
Bioengineering 
Biology 
Chemistry 
Engineering 
Geophysics 
Mathematics 
Physics 

Table 2 C 1 Subjects 1 academic majors by treatment groups 

ERLC 



Group 1 Group 2 Group 3 

2 0 3 

0 10 
0 2 1 

2 0 0 

110 
0 10 
111 
2 11 
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ERIC 



Group 1 Group 2 Group 3 

Junior 10 0 

Senior 111 

First year graduate 3 0 3 

Second year graduate 12 2 

Third year graduate 13 0 

Other 110 



Table 2» 2 Subjects 1 academic classifications by treatment 
"~ groups 



Group 1 Group 2 Group 



Semester hours credit 

(a) mean 18 0 5 15 a 6 13.6 

(b) range 12-30 3-40 6-24 

Cumulative average 

A 2 13 

B 4 5 2 

C 2 11 

Table 2a 3 Subjects 1 mathematics backgrounds by treatment 
groups 
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2o2*2 Motivation 

The experiment was presented to the subjects as a CAI 
project to determine the effectiveness of some new teaching 
materialsu The subject matter was related to , but not 
dependent upon, the students' current topic in the course, 
Emphasis was placed upon the consistency of the teaching 
program with the students 0 text in both content and notation u 
Students were asked to participate in a terminal session at 
their convenience in lieu of holding class on a particular 
day Q The pretest, given without notice, and the posttest, 
which was announced at the ti.ie of the pretest, were given 
during regular class tin?Jo The students were informed that 
their posttest score would be counted as one of approxi- 
mately 16 assignments but that no grades of any type would 
be given on the basis of their responses during the terminal 
session 0 It was felt that the above approach would provide 
sufficient motivation without inhibiting the students 0 
responses to the lesson c 

Students were not informed of the three output treat- 
ment groups until after the posttest was completed <, Instruc- 
tions concerning the disposition of the printed output were 
given by the teaching program itself , ao that no indication 
of differing treatments was presento 



2 0 3 Subject Matter 

2 0 3*1 Selection 

The choice of subject matter for the experiment was 
influenced in large measure by the envi ronment in which the 
study was to be conducted, The research was under the aus- 
pices of the Department of Computer and Information Science, 
whose students were readily available as subjects. It was 
felt that avoidance of the computer as subject matter would 
prevent potential confusion with the computer as a teaching 
device* The availability of a series of CAI programs in 
numerical analysis further influenced the decisiorio The 
series was designed as a semester course whose content and 
approach parallel that of the textbook used in a course 
offered by the Computer and Information Science Department 0 
The particular lesson was chosen because 

( 1 ) the subject seemed appropriate for the students 1 
expected progress by the date set for the experiment; 

(2) the professor did not intend to lecture on the 
particular topic in classy 

(3) the mathematical development of the lesson was 
relatively independent of other specific knowledge*, 

2 o 3 o 2 Content 

The objectives of the lesson were 

(1) to introduce the concept of the order of an 

k 

approximation and the notation 0(h ); 

2 

(2) to derive specific formulas for 0(h) and 0(h ) 



approximations to f'(x^); ^ 4 

2 

(3) to derive a specific formula for an 0(h ) approxi- 
mation to f " (x . ) , 

(4) to demonstrate the effect of roundoff error upon 
the foregoing approximations. 

The lesson was divided into the following seven sections, 
each dealing with one subtopic; 

A. Order of an Approximation 

B. Functions Tabulated on an Equally Spaced Set of 
Points 

CI, Order h Approximation to f'(x) 
2 

C2. Order h Approximation to f'(x) 
2 

C3, Order h Approximation to f"(Xg) and f'Kx^) 
2 

C4. Order h* Approximation to f'(x^), 0<i^N 
D. Computational Accuracy of Numerical Differentiation 
An outline of the material presented is included as Appendix 
A. 

Throughout the lesson, development of approximation for- 
mulas was based upon Taylor expansions without remainders, an 
approach slightly different from that of the students" text- 
book* All results and notational conventions were consist- 
ent with the text* Continual emphasis was placed upon the 
error in the approximations and the assumptions of con- 
tinuity and differentiability of the functions whose deriva- 
tives were to be approximatedo Section D gave students a 
chance to observe the 0(h z ) approximations to f'(x) and f"(x) 
as the interval size approached zero, and the effect of 
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roundoff error, 

2 » 4 Teaching Program 

2«4.1 Development 

The program was developed and tested by Dr fl Arthur E, 
Oldehoeft of Purdue University" 1 ' as part of a complete course 
in computational mathematics,, pursuant to a grant from the 
Office of Education , U 0 So Department of Health. v Education 
and Welfare [10 ] e A comparable course is taught at Purdue 
by conventional methods by Dr 0 S„ Do Conte,, whose textbook. 
Elementary Numerical Analysis [3], is the primary reference 
both of the programmed course and of COMP 150* The pro- 
grammed course, consisting of twenty- five lessons g was 
written in the language PICL3 (Purdue Instructional and Com- 
putational Learning System) g which operates interactively 
under the MACE Operating System on a CDC 6500 computer at 
Purdue,, Student terminals used for initial program testing 
at Purdue were KSR-33 Teietypeso 

A large portion of the development effort was devoted 
to writing and testing a function matching routine which 
allowed students to enter constructed mathematical expres- 
sions in response to questions * The matching routine 
checked the syntax of the student u s expressions and tested 
them for equivalence with expressions specified by the 



Now at Iowa State University,, 
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author. This allowed the student maximum flexibility in 
entering responses • The routine that performed this match- 
ing function was installed as part of the PiCLS system, not 
as part of this CAI program • 

The use of Teletype terminals restricted the notation 
to a strictly linear form, making lengthy expressions 
cumbersome. Such expressions were avoided where possible by 
use of multiple choice questions. Similarly , the use of 
graphs and diagrams was avoided because of the slow print 
speed and low resolution of the terminals. 

2.4.2 Local Adaptation 

Since the facilities available to this author did not 
include PICLS, extensive modification of the teaching pro- 
gram was necessary. For a variety of reasons, including 
availability, suitability to the task, and the investiga- 
tor^ personal preference, the program was translated into 
APL\360 [7]. Many of the vital portions of the original 
program were imbedded in the operating system rather than 
the program j comparable operation was provided through 
author-defined APL functions* No attempt was made to 
recreate the function matching routine because the particu- 
lar lesson being used required only three constructed- 
expression responses*. These were eliminated by changing 
the corresponding questions to a multiple-choice format* 
When a question called for a numeric response , however, 
students could enter expressions without reducing thenu For 
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example , applying the approximation, formula 

f (x ) - f (x. . ) 

D(h) = ii± where £(x. ,) * ,853, 

2h 14 1 

f(x^^^) = .603, and h = • the student could enter 

( .853- .603)*(2x ,i) , which would be evaluated and compared with 

the right answer* It was intended that this would relieve 

the student of performing the arithmetic by hand, but 

because of the students' lack of familiarity with the method 

of expression evaluation in APL, few students used the 

facility. 

Another problem which arose in the adaptation was that 
of capturing data on student responses v working time, etc* 
In the PICL3 version, such records were saved automatically 
by the system o In APL, system commands such as copying data 
into and saving data from an active workspace cannot be 
included in an author-defined function* It was therefore 
necessary to ask the student to perform a sequence of four 
or five system commands after eight of the nine sections of 
the lesson o Once again the students 0 lack cf familiarity 
with APL caused some data to be lost and some to be recorded 
incorrectly,, 

Since one of the treatment groups (See Section SUlol*) 
was not allowed to look back over the output ^ an attempt was 
made to eliminate the necessity of doing so* All tables and 
formulas were repeated where necessary to make them visible 
within the most recent 40 lines of output 0 
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Although extensive internal revision was done, the pro- 
gram remained basically unchanged in outward appearance. 
The advantages of a well planned lesson, written in colla- 
boration with the textbook author and tested in an actual 
course, were retained* 

2.4*3 Teaching Strategy 

2.4.3.1 General 

There were nine independent sections in the lessons, 
two of which (BEGIN and SUMMARY) were net directly concerned 
with presentation of subject matter* Six of the remaining 
sections, identified as PARTA, FARTB, PARTC1, PARTC2, PARTC3, 
PARTC4, used a linear teaching strategy in which the student 
was typically presented some text, then asked a question 
whose answer was a direct outgrowth of the text* The 
material vjas not presented in the small-step fashion often 
associated with programmed texts (e»g* "Fire engines are red # 
What color are fire engines? 1 '). Rather , making an intelli- 
gent response required completing a step in the derivation 
of a formula, applying a formula to a given set of data, or 
some comparable process. 

A correct student response brbught an acknowledgement 
of correctness and the next text or question. If a student 
entered an incorrect answer, he received a message, usually 
"NO. TRY AGAIN OR TYPE 'HELP ' • '* , and was asked the same 
question again, A right answer on the second try was 



29 

treated in the same manner as on the first try 0 A second 
wrong response, however, caused the correct answer to be 
printed, along with the reasoning leading to that answer « 

In lieu of an answer to the question posed, a student 
could type either "HELP" or "STOP"* The former was a request 
for a clue 0 The program responded with some message clarify* 
ing the question or suggesting an approach to finding the 
answer<> An entry of "HELP" was not counted as a response in 
determining when two wrong responses had been made 0 By 
typing "STOP" in response to a question, the student was 
able to leave the section in which he was workingo He could 
then sign off or begin any other parte 

2o4o 3 0 2 Functions ASKMC and ASK NUMERIC 

The logical sequence required to implement the above 
strategy wps contained in two functions, ASKMC and ASKNUMERIC 0 
Two routines were necessary because of the fundamental dif- 
ferences in the ways APL\360 handles character and numeric 
data* but the logical flow, as shown in Figure 2o4, was the 
same for both functions* The functions themselves are shown 
in Appendix B 0 

One of the two functions ASKMC or ASKNUMERIC was called 
by the main function (ice*, the lesson section) after some 
global variables had been set* (Global variables are those 
whose value is "known" both within and without the function 
being calledo ) The variables and their uses are as follows? 
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ir 



0 



Initialize 
indicators 
& counter 



Print 
QUESTION 



HELP ^sJ Yes A Print A 
requested?^ »( HELPMSG J — * 




Set stop 
indicator 



Record 
data 



Increment 

counter 




Record 
data 



/"Print 
-/ RONGMSG J 



Set help 
indicator 



Decrement 
Counter 




jReturn) 



Print. 
RTMSG 




Re tun 




Print 
TELLMSG 




Return! 



Figure 2*4 Common flow of functions ASKMC and ASKNUMERIC 



QUESTIONS A vector of characters used to pose the 

question. \_ 
RTMSG : Vector of characters; message to be printed when 

student responded correctly; usually "RIGHT". 
RONGMSG; Vector of characters; message to be printed 
when student responded incorrectly; usually 
"No, TRY AGAIN OR TYPE 'HELP'," 
HELPMSG 5 Vector of characters; message to be printed 

when student requested help. 
TELLMSGs Vector of characters; message to be printed 
after two incorrect responses; consisted of 
the correct answer and some expl anat ion. 
RT/vN,\) : either a number or o vocov .." r;i."Lorfl! 

the correct response, 
Note that RTANS may be either numeric or character t 
depending upon the question to be posed* The choice of 
function to be called was necessarily consistent with the 
type of data assigned to RTANS. 

Several differences existed between ASTCMC and ASKNUMERIC* 
The former was used primarily with multiple choice questions o 
In the case of only two possible choices s it seemed unwise 
to ask the question again after receiving one incorrect 
response „ The student would , in all probability g respond 
correctly the second time and proceed to the next question 
with no explanation of his error. A more satisfactory solu- 
tion would be to tell him he was incorrect 9 explain why, and 
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proceed with the lesson D To accomplish this, a parameter was 
passed to ASKMC giving the number of possible choices pre- 
sented to the student in QUESTION*, This number was used in 
initializing the counter which was used to determine when 
to tell the student the answer* The net result was that with 
two choices, one response was allowed, whereas with three 
choices, two responses were allowed* There were never more 
than three choices. For questions requiring numeric 
responses, the student was given the correct answer after 
two unsuccessful attempts* 

A second difference was the method of checking for a 
valid, or reasonable, response* No check is shown in Figure 
2.1 since the two functions handled the problem in very 
different ways* ASKNUMERIC was used only when the student 
was expected to respond with a number or an arithmetic ex- 
pression* APL\360 interpreted any alphabetic response as a 
variable to which no value had been assigned and printed the 
message "VALUE ERROR", followed by the student °s response* 
It was the author's opinion that this was sufficient explana- 
tion, so no other error-checking capability was included* 
In ASKMC, no system capabilities provided the error-detecting 
mechanism desired, so the function was designed to do some 
minor screening of answers* If the student 1 s response was 
neither "HELP 11 nor M STOP 11 nor one letter long, the student 
received the message M YOU MUST CHOOSE ONE OF THE GIVEN 
ANSWERS OR TYPE •HELP 1 • 11 A one-letter response which was 
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valid, such as "X" when the choices were "A' 1 , "B" 9 and "C" , 
was treated in the same manner as a valid, but incorrect 
answer. In neither function was an invalid response counted 
as an incorrect response for branching purposes. 

Another function, PAUS2, was called by both ASKMC and 
ASKNUMERIC, as well as by the lesson session main functions, 
to provide a variable length delay in the lesson to allow 
the student some "contemplation time"., The message "TYPE 
"P 1 TO PROCEED" was printed and execution was suspended 
until the student did so 0 PAUSE was used most regularly 
after TELLMSG was presented, allowing the student to analyze 
his incorrect responses and the correct one e PAUSE was also 
used after any long section of text„ particularly those with 
a substantial amount of mathematical notation Q A sample 
session, taken from PARTC4, is shown in Figure 2 0 5 0 The 
APLA360 program for the same portion is shown in Figure 2o6* 

2o4«3 d 3 PARTD Strategy 

The strategy employed in the last teaching section was 
markedly different from that of the other sections 0 It was 
an attempt to incorporate features of what Oldehoeft calls 
the Problem Mode and Investigation Mode* After several 
questions in the conventional mode fl the student realized 
that for f(x) = e x , fMx) = f"(x) = e x „ and that f»(0)= 
f "(0) = 1» Two approximation formulas which were derived 
earlier were presented to the student 0 One rbrumula^ 



SO OUR APPROXIMATION FORMULA FOR THE SECOND DERIVATIVE IS 
** 

** 02(J?)=(F[I+l]-2x?[I]+f[I-l])*(ff*2) 
** 

** F»'[I]-D2(ff)=-(fl*2)>«f ,, "(Z[I])*12, X[I-l]<Z[J]<Jf[I+l] 
** 

MEANS THAT D2(H)=0{WHAT?) 

A. H 

B. H*2 

CORRECT ANSWER IS : A ,B ? 
B 

RIGHT 

ASSUME THE FOLLOWING TABULATION FOR F(X)--(.X*5) + (2*X) 

I 0 12 3 4 

Kill -.10000 0 .10000 .20000 .30000 
FLU -.19999 0 .20001 .40032 ,60243 
FOR WHAT VALUE OF X DOES D2{H) NOT APPLY ? 

□ : 

0 > 

NO. TRY AGAIN OR TYPE 'HELP" . 

FOR WHAT VALUE OF X DOES D2(H) NOT APPLY ? 

□ : 

HELP 

TO CALCULATE D2(H) , YOU NEED VALUES FlI-ll, Fill, F[I+l], TRY AGAIN. 
FOR WHAT VALUE OF X DOES D2{H) NOT APPLY ? 

□ : 

.3 

RIGHT. D2 CANNOT BE APPLIED AT THE OTHER ENDPOINT , EITHER. 
SUPPOSE WE WISH TO APPROXIMATE F' * U[3]) . „ 
AT XL31, D2(H)=? 

□ : 

((.60243-2X. 40032)+. 20001 H. 01 
RIGHT, 02(3) = . 18.- 

THE EXACT VALUE IS F"(X[3]) = .16. NOTE THAT THE ERROR FORMULA IS 

F"[I]-Z?2(i?)=-(ff*2)xF" ' 1 (Z[I])vl2„ WHERE Xll-l ]<Z[I] <XU+1 1 . 
ON THE INTERVAL (Xl2l ) , 
MAXiABSW » » '(*)))=? 

□ : 

HELP 

WE NEED TO FIND THE MAXIMUM OF THE FOURTH DERIVATIVE OF 

F(X)=(X*5)+(2xX) . TRY AGAIN. 
MAXiABSiF 1 « » * (*)))=? 

□ : 



Figure 2.5_ Sample output from PARTC4 
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t (x. ,) - f(x. .) 

D(h) e - = — , was an approximation of f'(x. ). 

2h 1 

f (x. . ) - 2? (x. ) + f (x. , ) 
The other, D2(h) = — — - - - — " 



h 2 

approximated f M (x^). Each time the student typed "GO", the 
computer printed the values of h, D(h), and D2(h) for approxi- 
mations at as 0. On each iteration the previous value of 
h was multiplied by .1. Thus the student war* able to observe 
the convergence of D(h) and D2(h) to the true value of 1 as 
the interval size h decreased. After a number of iterations 
D(h) and D2(h) began to grow more inaccurate because of the 
roundoff error. The student had complete control of the 
number of iterations performed. 

A similar but more general mechanism followed. In this 
case the student was allowed to choose from a list of four 
functions, that function whose first and second derivatives 
were to be approximated. The four functions available were 

(a) f(x) * e x 

(b) f(x) = x 5 + 2x 

(c) f(x) s sin x 

(d) f(x) = 2(sin x - xcos x)o 

A fifth choice, labeled "NONE 1 ', permitted the student to 
terminate this part of the lesson. After choosing a func- 
tion, the student was asked to choose the point, x^ f at 
which he wished to approximate the derivatives. The value 
of x^ could be any rational number. For each function 
and point so chosen, the student could select any 



number of values of h, the interval size. For each h 
specified , the corresponding values of D(h) and D2(h) were 
printed. The student was free to choose any interval sizes 9 
perhaps strictly decreasing, perhaps not. It was hoped that 
students would explore the effect of roundoff error for 
various interval sizes at various approximation points. 
When the student typed "ENUF" , he was asked if he wanted to 
choose another x^ for the same function. If so, he was 
given that opportunity and could then specify new values of 
h. If not, he was presented the function list again, from 
which he could choose another function or "NONE". Figure 
2.7 shows one possible use of the features of PARTD. 

2.4. 4 Data Capturing 

The functions AS KMC and ASKNUMERIC described in Section 
2.4.3.2 recorded for each question whether HELP was re- 
quested and whether the student's answer was right or wrong. 
A question was counted as answered incorrectly only if the 
student never gave the correct response. Thus an incorrect 
response, followed by a request for HELP, followed by a 
correct response, was counted as being correct. These data 
were not kept separately for each question, but were added 
within each lesson section and recorded at the end of the 
section. Moreover, the time at which the student started 
the section and the elapsed time spent on that section were 
recorded. 
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Each student started the lesson by executing the func- 
tion BEGIN, which recorded some information, introduced the 
lesson, and gave directions regarding the printed output* 
Based upon the student's responses to several questions, the 
function recorded the student's location , the time of day, 
whether the student had used a computer terminal before, and 
whether he was familiar with APLo 

In the original PICLS program, this data collection was 
done by the system* In the APL\360 version used in the 
experiment, the data could be collected by the program but 
would have been lost if the system crashed or when the next 
lesson section was brought into the active workspace. As 
mentioned previously, there are APL\360 system commands 
which allow the user tc load, copy, and save date but they 
cannot be included in a function definition and executed 
later e It was therefore necessary to have the student 
execute the proper commands at the end of each section, The 
majority of the students did not know any APL and thus did 
not recognize failure of the system to perform the requested 
action* For this reason, some of the desired data were 
never recorded* Other data cannot be considered accurate 
because of untimely system failures* For example, if a 
student had progressed through a lesson section entering 
some correct and some incorrect responses, with an occasional 
request for help, and the system crashed just before he saved 
the data, there were two alternative courses of action upon 
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system restoration: 

(1) he could restart the section, in which case the 
response and timp ;"ata eventually r.nved would not 
accurately reflect his first pass through the section, 
or 

(2) he could start the next section, in which case no 
data would be saved • 

In a small-production, one-time session such as this, the 
data could be reconstructed through inspection of the output, 
but such a course is not practical in programs of larger 
scale* It is the author's feeling that as much data collec- 
tion as possible should be done automatically at the time of 
data generation. 

2.5 Equipment 

Two very similar types of terminals were used for the 
experiment, the Datel Thirty- 21 and the IBM 2741 * Moreover, 
two terminals of each type were used* Doth types of 
terminals look like electric typewriters with slight modifi- 
cations. The differences between the two types apparent to 
the user are listed in Table 2 0 8 0 



Datel Thirty- 21 



IBM 2741 



Rests on table like a typewriter 



Mounted in special table 



Electronic package attached at 
rear of typewriter 



Electronics mounted under 
table 



Three status lights 



No status indicators 



Outer cover blue-green 



Outer cover gray 



Table 2«8 Differences 



between terminals 
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The only difference which seems to have any potential 
influence upon the user is the absence of any indicators on 
the IBM 2741 as to the status of the terminal * The Datel 
terminals have three lights, labeled "READY" Q "PROCEED", and 
11 CHECK" o The READY light indicates that a connection exists 
between the terminal and the computer o PROCEED indicates 
that the keyboard is unlocked and the user can enter data* 
The CHECK light is illuminated when a parity error is found 
in transmissions to the terminal. When an error occurs the 
keyboard is locked and the CHECK light illuminated « There 
is a key, not found on the IBM 2741 Q which is used to free 
the terminal from this check status o 

An attempt was made to use the Datel terminals exclu- 
sively, but scheduling difficulties necessitated use of the 
two 2741" s« In general, students who did not finish the 
lesson in one session returned to the same location for all 
following sessions » The single exception completed 70% of 
the lesson at Location 1 and 30% at Location 2<, For data 
analysis, he is considered to have worked at Location 1 
exclusively^ As can be seen in Table 2 o 9 0 the majority of 
students used Terminal 1» Three of the four locations were 
judged to be not significantly different in physical condi- 
tions (i«e 9| lighting, heat v space, solitude)* Location 3 
was a small room which contained two other terminalSo It 
is quite possible that noise from the other terminals and 
their users proved distracting to the subjectSo 
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As explained earlier. Treatment Groups 1 and 2 operated 
the terminal in the normal manner. The paper fed continu- 
ously through the terminal and fell to the floor behind the 
table. A student could, at any time, pull the paper back and 
look at any part he wanted to see. Figure 2 o 10 shows the 
terminal as it appeared to these students. 

The objective with Group 3 was to simulate a CRT in 
terms of access to the output. A box-like device was con- 
structed to fit on the terminal so that the printed output 
passed through the terminal and into a slot in the box, 
allowing the user to see only the most recent part of the 
output. The effect is similar to using a CRT with "scrolling 11 
(i.e., displaying new data at the bottom of che screen and 
moving previous display up one line). The terminal as it 
appeared to Group 3 is shown in Figure 2 0 11. 



Location 



Terminal 



4 



Datel Datel 
Thirty-21 Thirty~21 IBM 2741 IBM 2741 



Subjects, Group 1 4 

Subjects, Group 2 3 

Subjects, Group 3 7 

Subjects, Total 14 



2 
2 
0 
4 



2 
1 
0 

3 



1 
0 
1 

2 



Table 2.9 Use of terminals 
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The amount of information displayed at one time is 
important to the CRT usero In this particular experiment it 
seemed that if members of Group 3 were indeed handicapped by 
the limited access to the paper that handicap would become 
more severe as the amount of output available was made 
smaller*, It was therefore decided that the amount of output 
accessible should be an upper bound of the amount actually 
shown on a CRT, A survey of CRT screen dimensions [l] led 
to a choice of 40 lines with 80 characters per line 0 While 
each cf these dimensions is provided on some CRT, no model 
known to the author provides botho The number of lines 
visible was limited by the distance from the terminal's type 
element to the slot in the appended box 0 The maximum number 
of characters per line was limited by the author in program- 
ming the lesson* 

2 0 6 Measurement of Treatment Effect 

The subjects were given a 30 minute pretest on the first 

day of the experiment o The pretest was explained to the 

student as a means of measuring their current knowledge of 

■ 

numerical differentiation so that the teaching program could 
be evaluated for effectiveness o No mention was made of the 
three treatment groups « In constructing the test, a distinc- 
tion was made between questions designed to measure the 
student's mathematical background and those designed to 
measure the student *s specific knowlege of numerical 
dif f erentiation. The former will hereafter be referred to 
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collectively as Pretest l fl the latter, Pretest 2, The inten- 
tion was to perform analyses of covariance using each of the 
pretests as covariate to determine the effect of the various 
treatments . On the actual test, a copy of which is Appendix 
C B questions numbered 1 v 3, 5 0 6 formed Pretest 1* questions 
numbered 2, 4 ( 7 formed Pretest 2 0 A moderate positive corre- 
lation was found to exist between the scores on each pretest 
and those on the posttest and between the scores on the two 
pretests , as shown in. Table 2*12* 

Pretest 2 Posttest 
Pretest 1 0 4224 0 3264 

Pretest 2 - «4681 

Table 2 o 1 2 Coefficients of correlation between 
scores on pairs of tests 

The questions on the posttest, which was to be completed 
within 50 minutes 0 were based directly upon the presentation 
made to the students at the terminal 0 All questions contri- 
buted equally to the total score c After the posttest was 
given 9 a questionnaire on which the students were asked to 
record their reactions to the lesson was distributed* Copies 
of the posttest and questionnaire can be found in Appendices 
D and E, respectively <> 

Construction and scoring of the tests t as well as con- 
struction of the questionnaire 9 were done by the author* The 
students' names on the papers were replaced by numbers by 
an impartial observer before the papers were graded, thereby 
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eliminating the grader's bias« Independent numbering schemes 
were used for the pretest and posttest to ensure the absence 
of grader prejudice. 

2.7 Other Design Features 

2.7.1 Proctors 

Each student was observed during his terminal session 
by a graduate student from the Department of Computer and 
Information Science. These proctors were present to ensure 
that the experimental conditions were adhered to and to 
assist the student with system difficulties. In particular, 
they placed the box on the terminal 0 collected output 9 
assisted with initial sign-on^ ensured that the data-storing 
and program-loading system commands were properly executed, 
assisted the student with communications failures, and 
recorded any observations they felt significant 

Although the presence of a proctor was desirable in this 
controlled experiment Q it may not be the best arrangement,, 
On the questionnaire 0 15% of the students responded that they 
were "slightly bothered" by the proctor 0 a presence. 

2.7.2 Realistic Setting 

The experiment was conducted in an active academic envi- 
ronment, not an artifical, experimental situation c This 
11 real world" approach to the study created some uncontrolled 
variables such as the amount of use the students made of the 



textbook 0 the amount of study time per student „ and the 
possibility of members of Groups 2 and 3 borrowing output 
from members of Group 1. However, it was felt that for any 
results to be applicable to a realistic problem g they must 
be obtained in a realistic situationo In keeping with this 
philosophy, the students were referred to Ccnte [3] for ^re- 
lesson reading. Those who did not have a copy of the session 
output were at liberty to study the same or any other text 
after the lesson. 

2.7„3 Placebo Effect 

In experiments of this nature it is sometimes the case 
that one group performs better than others because they 
think they are receiving the superior treatment 0 In order 
to reduce this placebo effect it is necessary to keep the 
treatments unknown to the subjects. The proctors made every 
attempt to have the terminal attachment in place before mem- 
bers of Group 3 arrived and to remove it before other sub- 
jects entered the room. The instructions for disposing of 
the output were printed by the program , based upon previously 
stored data on group membership 0 It is not possible to deter- 
mine how much information was relayed between groups in 
informal discussions 0 

2o7 0 4 Time Span 

The pretest was given on a Wednesday 9 at which time the 
students chose the times for their terminal sessions. Approx- 
imately 22% of the sessions were held on the following two 
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days, with the remainder being conducted on Monday Q Tuesday 9 
and Wednesday of the following week* The posttest was given 
nine days after the pretest, with the time lapse between 
lesson and posttest ranging from two to eight days 0 

2*7 0 5 Group Assignment 

Students were assigned to treatment groups using the 

APL\360 function "deal" o A vector was generated, each 

component of which was a random number from one to three 0 

The alphabetic class roll was matched with this vector to 

assign treatments* When the absence of three students from 

the pretest produced unbalanced groups, a number n was 

chosen at random from the integers 1 through 7„ The alpha- 
th 

betically n member of Group 2 was changed to Group 3* 



Chapter III 
THE RESULTS 

3* 1 Comparison of Treatment. Groups 

3ol 0 l Posttest Scores 

An analysis of covariance between the three treatment 
groups was performed using the two pretest scores as co- 
variates and the posttest scores as the dependent variable * 
The results of the analysis are summarized in Table 3.1 0 
There is no evidence to support rejection of the null hypo- 
thesis that the three treatment group means are equal when 
pretest differences are controlled*. 

Sum of Sum of Sum of 

Squares Squares Squares Mean 

Source df (y ) (within ) (between) df Square 

Treatment 2 217 e 375 202.254 15.121 2 7.560 

Error 17 1757*188 270 0 140 1467.047 15 99*137 

Total 19 1974,563 472.395 1502.168 17 

F = o076, not significant 

Table 3ol Analysis of covariance of posttest scores using 
pretest scores as covariates* 

Table 3*2 shows the mean score for each group on each 

measure* Notice that Group 2 scored consistently lower on 

all tests, perhaps for reasons cited in Section 2 a 2<>l« A 

complete tabulation of test scores is found in Appendix P, 
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Pretest 1 Pretest 2 Posttest 

Group 1 42o57 33,86 86,50 

Group 2 31.33 24,50 77.17 

Group 3 39.57 31,43 82,88 

Table 3. 2 Test score means by treatment groups 



Sum of 

Source Squares df Square F~Ratio £ 

Question 1 

Treatment 191.-93- 2 95,97 3,6U p> .05 

Error 479,88 18 26,66 

Question 2 

Treatment 46,91 2 23.45 0 72 NS 

Error 585 0 76 18 32.54 

Question 3 

Treatment 11,03 2 5. 51 , 79 NS 

Error 125,92 16 6,60 

Question 4 

Treatment 16,60 2 8,30 1.01 NS 

Error 148.36 18 8.24 

Question 5 

Treatment 6,40 2 3,20 ,19 NS 

Error 298.55 18 16,59 

Table 3,3 Analysis of variance between treatment groups by 
posttest question, 

A simple analysis of variance was performed to determine 
whether the group scored significantly differently on individ- 
ual posttest questions. The results are summarized in Table 
3,3. Notice that no significant difference was found except 
on Question 1, which reads , M Explain what it means for D(h) 
to be an 0(h ) approximation to a number a. 11 With the ques- 
tion being graded on a 20-point scale g the mean scores were 
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as follows: 

Group 1 14. 375 

Group 2 3,000 

Group 3 15,000 
The significance of the F value seems attributable to the 
low scores of Group 2, If the low scores were due to lack 
of hard copy to study after the session, one would expect 
low scores from Group 3 as well* In the absence of such low 
scores, one suspects that the difference is not due to the 
treatment o Accordingly, an analysis of covariance was per- 
formed using the two pretest scores as tnultiple covariates 
and the scores on Question 1 as dependent variable*' The 
result, F (2,15) = 1*802, is well within the range of non- 
significance at the .05 level, 

3ol#2 Lesson Performance 

In all instruction, and especially in CAI , a desirable 
goal is to achieve maximum learning in a minimum amount of 
time* Thus if two methods of teaching produce the same re- 
sults, but one method does so in significantly less time, 
then it is judged to be more desirable* With this in mind, 
an analysis of variance was performed on the total working 
time of the three groupsa The times used were adjusted to 
exclude time consumed by system failures* Although the 
difference was not significant, it is interesting to note 
that Group 3 required an average of 10 minutes less than 
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any other §roup„ 

Ju r i ng the terminal sessi on 4 Groups 1 and 2 had the 
same access to the printed output while Group 3 was re- 
stricted. It therefore seems reasonable to compare Group 3 
with the union of Group 1 and Group 2 to determine the 
effect of limiting access to the outpuio One notes from 
Table 3*4 that no difference was observed in the number of 
requests for help or in the number of wrong responses, (A 
wrong response is here defined as a question to which the 
student did not give the correct answer in the maximum 
allowable number of tries,, ) Again, there is an apparent, 
although statistically insignificant, difference in com- 
pletion times. The relatively high number of wrong 
responses and requests for help by Group 2 are consistent 
with that group* s relatively poor scores on all three testSo 

Grou p 1 Crou;j 2 Group 3 ^roup 1 and 2 

i ] o a n c oiii p 1 e t i on 

ti-;ie in minutes 124 12 'J 114 12G 
Mean number of 

wrong responses 5.3 7*4 C fl 7 6.4 
Mean number of 

requests for help 3 0 7 5*1 4.9 4,4 

Table 3 e 4 Group means for completion times g wrong responses, 
™ — anc j requests for help 

3 0 i,3 Qu es t i onnai re Re sponses 

In determining the use made of the output while at the 
terminal. Group 3 should be compared with the union, of Group 
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1 and Group 2 Q Of the six members of Group 3 who returned 
questionnaires, only two were more than "slightly frustrated" 
at not being able to see the outputs On the other hand, 
60% of Groups 1 and 2 said they often referred to the por- 
tion of the output that would have been hidden from Group 3, 
This suggests that having the output may be useful , but that 
Group 3, having never developed the habit of looking at the 
output, did not miss ito Indeed, the ability to look back 
was judged "considerably useful" or "very useful" by 87% 
of Groups 1 and 2* 

With regard to output use after the session, Groups 2 
and 3 may be combined. Perhaps the most significant point 
to arise from the questionnaire is that the students who 
had no session output studied the textbook instead*, The 
total amount of study time was approximately the same for 
the two conditions but Group 1 studied the session output 
almost exclusively o Of this group, 70% ranked the output 
more useful for later study than the textbook* 

Because of the printing method employed in most type- 
writer terminals, a linear notation must be used to achieve 
reasonable efficiency*. Combined with a lack of lower case 

f (x- . - ) « fix. n ) 

letters, this causes the formula D(h) = - — ±£± x ~ ± 

8 2h 

to be printed as D(H) = (F (Z[J+l])-F(A:[J-l]))-5-(2xi?) e All three 
treatment groups reported experiencing "moderate" to "con- 
siderable" difficulty reading such notation . 

ERLC 
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3 • 2 Other Comparisons 

In addition to comparisons of treatment groups, several 
questions arise which are interesting in their own right and 
thej answers to which serve as validation for the experiment* 

The . results of these comparisons, all of which are simple 

/ . — ~-- / 

analyses of variance among posttest scores, are summarized 
in Table 3.5 and explained below. 

Basis for Grouping df jp Critical F Experimental F 

Day of terminal session 1,19 NS 4„ 38 3.53 

Terminal location 3,17 NS 3.20 2,61 

Type of terminal 1,19 NS 4o 38 1.60 

Experience with 

terminals 1,18 NS 4 0 41 0.16 

Table 3o 5 Degrees of freedom, significance thresholds, and 
F ratios for miscellaneous comparisons of post- 
test scores 

The terminal sessions ranged over a seven day period , 
with the posttest coming two days after the last session*. 
It seemed possible that those students who had completed the 
lesson most recently would do better on the posttest becauset 
of the time difference 0 The students were divided into two 
groups (n^ = 10, n 2 = according to the day of their 

terminal session* These two chronological groups were com- 
pared on the basis of posttest scores. There was no signi- 
ficant difference in scores. Moreover, the difference that 
did exist , seven points on a 100-point scale, was in favor 



of the chronologically first group 0 Since the students chose 
their own session times, this result may reflect a tendency 
of the poorer students to postpone the session as much as 
possible o 

Pour terminals, in four locations, were used for the 
experiments Two of these were Datel Thirty~21 9 s and two 
were IBM 2741 0 s o Differences in external conditions such 
as lighting e noise, temperature, and other nearby activity 
have potential influence upon the students 8 retention of 
subject matter o The appearance and functions of the two 
types of terminals may possess similar potential 0 Analyses 
of variance of posttest scores using both terminal type and 
terminal location as grouping criteria were performedo In 
neither case was a significant difference found* 

Finally, a comparison was made to determine whether 
students who had never used a terminal before suffered from 
a form of "stage f right" 0 Although they may have felt ill- 
at~ease during the lesson , these first- time users did not 
score differently on the posttest* 

3 0 3 Summary 

3o3ol Efficiency of Learning 

Neither the use of printed output during the terminal 
session nor its availability after the session had a signif- 
icant effect on posttest scores when college students were 
taught numerical differentiation through computer- ass is ted 
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instruction* Those students who had access to the paper 
during the session took 10% more time to complete the lesson 9 
an observable but statistically insignificant difference* 
Access to the output was not found to affect the correctness 
of responses or the number of times the students requested 
clues during the lesson 0 

3 0 3o2 Student j^titudes 

Those students who had access to the output during the 
session felt that it was very useful 0 while those who had 
no access wei*e not particularly bothered by that facto Those 
who received the paper after the terminal session used it 
extensively as a study guide & Those not receiving the out- 
put reported that they would have studied it, had it. been 
avail able o Nearly all of the students without the printed 
output compensated for its absence by studying related 
material in the textbooko 
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Chapter IV 
THE CONCLUSIONS 

4 0 1 Apparent Implication^ 

Under the conditions of this experiment and subject to 
the limitations discussed in the following section, it 
appears that no ill effects would be suffered from use of 
a terminal which does not produce hard copy output* This 
is equally true when the quality of learning is measured 
by the correctness of student responses during the session 
and when it is measured by post test scoreso 

If hard copy were universally insignificant then for 
those people who were about to establish, expand, or upgrade 
a CAI system, the CRT would become clearly more desirable 
than the typewriter terminal 0 One could then have the speed 
and flexibility of the CRT without having to pay for a 
separate printer or penalize the student by not providing 
hard copy 0 For those already committed to CRT°s, programming 
complexity would be reduced by elimination of the need for 
making provisions for hard copy 0 

Students with access to the printed output during the 
terminal session (Groups 1 and 2) took slightly longer to 
complete the lesson 0 This difference may be a result of 
the students u looking back at previous output which was 



not available to Group 3 0 The fact that Group 3 learned 
as much in less time seems to indicate that the extra time 
was not used effectively 

The most obvious use of hard copy output is as a refer- 
ence for further study Q As would be expected^ those students 
who were allowed to take the output with them after the 
session used it extensively as a study guide <, Those not 
receiving the output reported that they would have studied 
it, had it been available*. Although this post-lesson " study 
may not occur in all CAI situations 0 it was certainly not 
unexpected in this particular instance* 

It is noteworthy 0 in the author's opinion 0 that those 
students who did not receive the output spent as much time 
in post-lesson study as the group receiving the output 0 
compensating for the lack of hard copy by studying the text- 
booko Certainly, many factors in the experimental design,, 
notably the availability of a substitute source (the text- 
book) and the presumed h igh motivation of the students Q 
contributed to this phenomenon Nevertheless, there is 
evidence here that students can and will compensate for 
the lack of hard copy 0 It may even be the case that this 
forced exposure to two different sources contributes to a 
broader understanding of the subject matter.* 

Difficulties in reading the linear notation imposed 
by the typewriter were equally severe for all treatment 
groups*, The problem was especially acute in this instance 



because it was the students 1 first (and only) lesson using 
that notation. Although students may experience less 
difficulty as their exposure to linear notation increases, 
the notation remains inconsistent with standard mathematical 
usage and therefore incommodious. 

4. 2 Limitation on Generalization 

Certain limitations are imposed upon the generality 
of any experimental results by the circumstances under which 
the study was made* The experimental design and the ration- 
ale thereof are discussed in Chapter III; this section con- 
siders the applicability of the results of this study to 
other situations. 

The value of hard copy output is certainly dependent 
upon the learning objectives established by, or in conjunc- 
tion with, the author of the CAI program. The objectives, 
of course, determine the content of the lesson and the 
desired detail of recall. The desired outcomes of the 
lesson used in this study were understanding of concepts 
and ability to apply formulas to specific data. It seems 
reasonable that hard copy would be more valuable as a study 
guide if specific formulas were to be memorized or if one 
were learning, for instance, state capitals. 

The tests used in this investigation reflect the objec- 
tives of the lesson. Each of the five questions on the 
posttest had a value of twenty points* Partial credit was 
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given for partially correct answers. It is possible, 
although unlikely in this case, that the type of test 
affects the variance among treatment groups. 

Also related to the desired outcome of instruction is 
the mode of instruction used. The role of hard copy out- 
put may be quite different in simulation and student-con- 
trolled problem-solving modes from the role in author-con- 
trolled mode. No attempt has been made to measure the 
value of hard copy other than in computer-administered .pro-r, 
grammed instruction; no attempt should be made to generalize 
these results to any other mode* 

It is worth noting here that the students who received 
no hard copy output to take away from the terminal did have 
access to a source of similar information f the textbook* 
When the computer is used in modes other than author-con- 
trolled, there may not be such a comparable source, in which 
case the value of hard copy output may well increase . 

The study reported herein is based upon one lesson and 
a test that followed within ten days e It is not difficult 
to imagine that a copy of the output may become more valuable 
as a study guide as the time between lesson and test in- 
creases. It is not clear how students' use of hard copy 
output would change if a series of lessons were used instead 
of just one. 

Perhaps the most important factor one must consider in 
generalizing from these results is the characteristics of 
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the studentSo The subjects used in this ifv/estigation were 
of above average intelligence, well motivated, mature, and 
well accustomed to the academic environment <, When confronted 
with the lack of hard copy output they had the ability and 
the initiative to compensate,, Obviously g one would not 
expect all levels of students to react in the same manner*, 

To the author's knowledge 9 this is the only investiga- 
tion that has been made to determine the value of hard copy 
output o It should be regarded as a study of one particular 
CAI situation; generalization should be made only in the 
light of the preceding discussion * 

Further investigations should proceed along several 
lines in an attempt to determine the circumstances under 
which hard copy output is beneficial to the student 0 The 
most obvious area of interest is that of student character- 
istics „ particularly age c Other modes of computer use 
should be investigated as well as other subject matter „ 
To ensure validity, future studies should use larger treat- 
ment groups for extended periods of timeo 
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APPENDIX A 

LESSON OUTLINE 



I. PARTA Order of an approximation 
A 0 Define D(h) to be "order h" if 

lim / A ~DOi) ) ^ ^ 

h ^ q I — H — J = a nonzero constant, 

B. From Taylor's expansion of f(x Q + h) about Xq, show 

that D(h> = f (x 0 * h ) ~ f (x 0 } is of order h. 

h 

C. From Taylor* s expansions of f (x^ + hi and f(x^ - h) 

about Xq, show that 

f (x* + h) - f(x n - h) ~ 

D(h) = u — — is of ord*r h . 

2h 

II. PARTB Functions tabulated on an equally spaced set of 

points 

A. Introduce tabular format;; note equal intervals. 

B. Apply the two approximations developed in PARTA to 
one point in a given table* 

III. PARTC1 0(h) approximation to f ( (x) 

A. Given a table and the analytic form of the function, 
find the maximum error o 

B. Apply the 0(h) approximation formula at three dif- 
ferent points. 

C. Compare each approximation with the true value. 

D. Compare each error with the maximum error. 

9 

ERLC 
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2 

IV. PARTC2 0(h ) approximation to f'(x) 

A. Note that D(h) = f * x i + 1 j ~ f * X i - 1 * cannot be 

2h 

used at the ends of a tabulated interval. 

B. Find the maximum error for this formula in the 
given interval. 

C. Apply the formula at the same points used in PARTC1. 

2 

D. Compare the 0(h) 9 0(h ) and true values at the three 
points. 

V. PARTC3 0(h ) approximation to f»(x 0 ) and f^Xjj) 

A. From Taylor's expansions of f(Xg), ftx^) and ft^) 

about Xq, derive 

-3f (x n ) + 4f (x. ) ~ f (x-) 
D(h) = 0 — 



2h 

B. Find the maximum error in the interval. 

C. Apply the formula and note that the error falls 
within the predicted bounds. 

D. Do the same sequence for fCx^). 

VI. PARTC4 0(h 2 ) approximation to fMx^, 1 < i< N 

A. From Taylor's expansions of f(x^), f(x^ ^) and 

f(x i _^) # derive 

f (x. , ) - 2f (x. ) + f (x. . ) 

D2(h) = 1+1 ' ^ • 

h^ 

B. Find the maximum error in the interval. 

C. Apply the formula and note that the error falls 
within the predicted bounds. 



9 

ERLC 
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VII. PART D Computational accuracy of numerical differen- 
tiation 

f (x. A , ) - f (x. . ) ari - 
A. Using D(h) t-±-i and 



9 

ERIC 



2h 

f (x, ^ , ) - 2f (x. ) + f (x. . ) 
D2(h) = ^_±_J= i 1 - J- ^ show that 



h' 

error approaches zero as h approaches zero. 
B. Show roundoff error in approximations to f'(x) and 

f" (x) for f(x) =s e by letting h approach zero. 
C # Present student with functions and mechanism for 

observing roundoff error under student control. 



APPENDIX B 
FUNCTIONS ASKMC AND ASKNUMERIC 



V ASKNUMERIC 

[I] COUNT+FLG+HLPFLG+O 
[2] QU: QUESTION 

[3] HHELPX ,STOPX , ( ( pETANS ) pETX) ,EONGX ) [ (HELP ,STOP ,ETANS ) \ANS+Ul 
M HELPX: -+QU,( pQ+HELPMSG) ,HLPFLG+1 
[5] STOPX:+(FLG<-1)/0 

[6] ETX: , ( pQ+ETMSG) , ETEEC+ETEEC+ 1 ,HLPFLG 
C 7 ] EONGXi C0UNT+C0UNT+1 
[8] +(C0UNT>2)/ENDX 
[9] ->QU ,pD+-EONGMSG 
[10] ENDX-.TELLMSG 

[II] EONGEEC+-EVNGEEC+ 1 ,HLPFLG 
[12] P4£/SP 

V 



[I] C0UNT+3-NUM 
[2] HLPFLG+FLG+O 
[3] QU -.QUESTION 

^(^pMS+QVPOi/PLPTS 

[5] ->cia: 

[ 6 ] FOUELETS : -* ( FLG+ANS* . -- ' STOP ' ) / 0 

[ 7 ] -*-(ANS^, - ' #PLP ' ) /#PLPX 

[8] imLi-^i/.pO'ioi/ most cwoasp cm op rap givew answers oe 

TYPE ' '#PLP' » . ' 
[9] HELPX-.+QU ,(p\yHELPMSG) ,HLPFLG+1 
[10J OUf:+(l*p4/VS)/ItfK4L 

[II] C0UNT+C0UNT+1 
[12] HANS=ETANS)/ETX 
[13] -> ( C0UNT>2 ) /ENDX 
[14] -*QU ,pU*-EONGMSG 

[15] PTX : -K) , ( pOPJMSG ) , ETEEC+-ETEEC+ 1 ,HLPFLG 
[16] ENDX-.TELLMSG 
[17] R0NGEEC+-E0NGEEC+ 1 , flLPPLG 
[18] P40SP 
V 



APPENDIX C 
P RE -EXAMTNATI ON 



COMP 150 



October 21, 1970 



1, Give an expression for the slope of the line shown below. 




2« Explain the relationship between discretization error and roundoff error. 

2 

3. Given f(x) = 2x 5 - 7x 3 + 4, evaluate lim f f TIT (2+a)J . 

a->0 V 5 



4o Given a tabulated set of values for x^ and f(x^), i=l,2, a .*,8, why can 

D(h) = ^i+1 ^i-1 not be applied at i=8 ? 
2h 

5. What is the geometric significance of the first derivative of a function ? 

6o State Taylor's formula with remainder for expansion of f (x) about a point, a, 

7. Suppose you are given the graph below and the coordinates of the points 

a,b,c,d. How would you estimate the value of the derivative of f evaluated 
at x, ? ^ 



APPENDIX D 
POST-EXAMINATTON 

COMP 150 October 30, 1970 

NUMERICAL DIFFERENTIATION 

k 

le Explain what it means for D(h) to be an 0<h ) approximation to a number a 0 
2 e Given the Taylor expansions 

f i + l = f i + hf i' + T f i" +| 3 f 1 "(-) ^ere x.<w<x i+1 

2 3 
f . = f - hf + 4 f .' • ■- ~ 



"i-1 1 



. ' + $ f. ,! - ~ f'"(v) where x. 1 <v<x. , 

1 Z 1 0 1-1 1 1 



f. , - f. , 

derive the formula for the error in the approximation D(h) = — - — ^ — ^~ " 
Of what order is this approximation ? 

3. The following are all approximations to f.', 'What are the advantages and 

disadvantages of each ? That is, under what circumstances would each be used 
f,.. - f. 

(a) D(h) = - i± V~ i 

f.,, - f. , 

(b) D(h) = 

-3f. + 4f . . - f.,_ 

z \ n/ , ■, " ' i l+l i+2 

(c) D(h) = — 2h — - 



4c Given the following cable, apply at x^ each of these approximations < 
f.^T - f. 1 f. . - 2f. + £... 

•n/,^ _ 1+1 1-1 n /i \ 1-1 1 1+1 

D(h) - 2h D(h) = —J— 

n 

i 0 12 3 4 

x. 1.2 1.3 1.4 1.5 1,6 

i 

f. 1,510 1,698 1,904 2.129 2.376 



5; In the context of numerical differentiation, explain what is meant by the 
"optimal value of h". 



APPENDIX 



POST-EXAMINATION QUESTIONNAIRE 



COMP 150 Computer-Assisted Instruction Project 

Attitudinal Survey October 30, 1970 

The answers to tne following questions will be used in conjunction 
with your test scores in attempting to draw some conclusions regarding 
the value of the printed output in computer-assisted instruction. While 
inclusion of your name would be helpful in comparing reactions to per- 
formance f you should feel free to return this form lansigned if you pre- 
fer« 

A a .1 o How many semester hours of mathematics have you taken? 

2* Circle the best estimate of your cumulative math grades. 
A B C D F 

3. What is your major field? i 

4. Circle your classification* 

Jr, Sr. Grad 1 Grad 2 Grad 3 Other 

B, Please circle the word or group of words which best describes your 
leelmqs or actions. 

1. Before the terminal session t I studied the appropriate section 
of Conte 

none less than 15 min. 15-30 mine 30-60 min. more than 60 min. 

2. After the terminal session, I studied the appropriate section 
of Conte 

none less than 15 min. 15-30 min. 30-60 min. more than 60 min. 

3. I had trouble reading the linear notation ('subscripts, exponents, 
fractions) . 

not at all slightly moderately considerably very much 

4. difficulties with the terminal 0 telephone ff and computer inter- 
fered with my concentration on the material presented. 

not at all slightly moderately considerably very much 

5. The presence of the proctor bothered me. 

not at all slightly moderately considerably very much 

6. The clues presented when I typed "HELP" u-xre actually helpful 
in understanding the material. 

never seldom occasionally usually almost always Never typed 

"HELP" 
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lo I purposely typed a wrong answer just to see what would 1 happen, 
never seldom occasionally usually almost always 

8 0 I asked for help when I did not really need it c 

never seldom occasionally usually almost always 

9* Trying to put mathematical expressions in the proper format 
interfered with my concentration on the material presentedo 

not at all slightly moderately cpnsiderably 

very much didn D t enter expressions 

There were three different conditions in the experiment described 
below as C, D r E Q Circle the letter corresponding to the group you 
were in and answer the associated questions** 

Co A box-like device ©n the terminal limited my access to the paper o 

lo I felt frustrated at not being able to see the previous outputo 

not at all slightly moderately considerably very much 

2o If I had been allowed to take the paper along , I would have 
studied it later c 

no maybe probably surely 

3 0 I borrowed some one else°s paper to study 0 
yes no 

4 0 During the terminal session, I pulled the paper out of the slot 
in order to read something 0 

never seldom often 

Do I could look at the paper during thf* terminal session, but was not 
allowed to take it from the room c 

lo I referred to the printed output (more than one foot back) 0 

never seldom often 

2c If I had been allowed to take the paper along, I would ha r /e 
studied it later 0 

no maybe probably surely 

q 3 0 I borrowed some o,. s else's paper to study Q 

ERIC yes 
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4o Being able to look back at the paper was useful,, 

not at all slightly moderately considerably very much 

I was allowed to refer to the printed output while at the terminal 
and to take it along after the session. 

1, l roiorred to the printed output (more than one fool; back). 

novex seldom often 

2* Being able to look back was useful. 

not at all slightly moderately considerably very much 

3. After the terminal session, how much time did you spend 
studying the output? 

4. Being allowed to take the paper along was useful, 

not at all slightly moderately considerably very much 

5. I found the following more useful for later study. 
Conte session output about equal 



APPENDIX F 



EXAMINATION SCORES 



Pretest 1 Pretest 2 Postte 

Group 1 47 40 100 

98 

30 25 94 

50 40 92 

42 45 87 

35 37 03 

45 25 71 
49 25 67 

Group 2 - 50 25 09 

15 10 ,,2 

25 32 80 

39 32 76 

34 20 72 

25 20 64 

30 20 

Group 3 49 28 96 

46 43 93 
23 25 04 
42 32 83 
46 22 82 
41 40 80 

77 

30 30 68 



